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communication system 



(57) The communication device comprises a keypad which includes an array of input transducers 20-26 and 
tactile buttons which include output transducers 10-16. The device makes text, graphics and speech, held on a 
computer or communicated via a computer, accessible through the sense of touch. 

For text, the device displays one character at a time using the array of buttons 10-16 to tap or vibrate 
against the skin and produce a pattern of sensation on the skin. For graphics such as maps, the device allows 
the user to explore in 2-dimensions, and use the sense of touch to locate objects on the map. For speech, the 
device displays phonetic patterns based on output from an automatic speech recognition system. 

The system combines the keypad 20-26 with the tactile output 1-16 into a single hand-held unit. 
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TACTILE SYSTEM FOR COMPUTER DYNAMIC DISPLAY AND 

COMMUNICATION 

This invention relates to a tactile system to make text, graphics and speech, held on a 
computer or communicated via a computer, accessible through the sense of touch. A 
text-based version of the system is intended for use by blind and deafblind people, but 
could also be useful for deaf people, or for people with processing impairments such as 
dyslexia. A speech-based version of the system is intended for people with 
communication problems because of deafness, speech impediment or both. 

Most blind people cannot read Braille. Learning to read Braille is difficult for most 
people, especially elderly people, partly because of the fine sensitivity of feeling 
required in the finger tips, and only a small proportion of blind people learn to read 
Braille fluently. Alternatives to Braille have been tried, for example the systems known 
as Bliss and Moon, but with limited success. Special printers and paper are needed to 
produce hard-copies of such notations, and this gives rise to high cost with small 
quantity production. 

Braille books are expensive to produce, bulky to store, and heavy to carry around. But 
with the advent of light-weight portable computers, information in electronic form can 
be carried around easily. Unfortunately dynamic Braille displays are bulky, expensive 
and power consumptive. A known dynamic Braille display comprises an array of cells, 
each displaying a single character. The cost per cell is considerable and it is multiplied 
by the number of cells in the array, typically 40 or 80. A few systems have been 
developed with a single Braille cell, but though the cost is reduced, the problem of 
finger sensitivity is aggravated, and blind people complain of numbness after using such 
a device for a few minutes. 

Although voice synthesis often cm be used with Braille in place of tactile output, there 
are many situations when sound is obviously undesirable, and tactile output by itself 
preferable, for example in the classroom. Blind people prefer Braille for reading 
documents containing technical jargon or numerical data. Many blind people prefer 
Braille for a quiet read of a book, when they can leave the sounds to their own 
imagination. Therefore tactile output is an extremely desirable supplement to sound. 
However the advent of good-quality, affordable speech synthesisers is making it 
increasingly difficult for manufacturers to sell and maintain expensive tactile devices, 
for example production of the system known as Optacon (see below) will be 
discontinued at the end of 1996. 

Blind people have problems accessing graphical information. Blind people have a 
problem to access any kind of information in graphical form. Tactile graphic "hard- 
copy" images can be produced, e.g. using swell paper or moulded plastic, but there are 
a number of disadvantages: 

• the images are static; 

• they are low resolution - with limited information content, 

• they are not scaleable; 

• they require special expensive equipment to be produced; 

• each image has a production cost per copy; 
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• the image is bulky, and unwieldy to cany around; 

• an image once produced cannot be easily marked or annotated. 

The main advantage is that there is no cost associated with actually reading the image; 
the user merely scans with a finger over the surface. 

The information content can be boosted by having audio output of text annotations, and 
mounting the image on a touch-sensitive tablet. This is the approach of the system 
known as Nomad. However other disadvantages remain, and there is now a cost 
associated with the reading system. 

A dynamic display should not have any of the disadvantages of a purely hard-copy 
image, but hitherto the cost of dynamic display systems has been a problem, especially 
for a system with reasonable resolution. There have been three approaches tried. In 
the first approach, that of the Optacon, there is a array of a large number of small pins, 
each pin corresponding to a pixel. The pins act over the surface of a finger-tip. The 
device is mounted on a mouse to allow a pointer (cursor) to be moved over the graphic 
image. Fine finger sensitivity is needed, and the solution is expensive because of the 
number of pins involved, each operated by a separate piezo-electric device A second 
approach has been to use a virtual reality display with a tactile glove incorporating 
position sensors and actuators. This has also proved very expensive. In a third and 
recent approach, that of the device known as FeelMouse, the button on a mouse has 
been modified with tactile feedback, such that the user can feel a force related to what 
is "under" the cursor on the screen (assuming there is one for the sake of discussion). 
This is a much cheaper solution, but provides only limited sensory input to the user - 
-that is, the channel for information flow to the user is very narrow, and the user would 
require a long time to build up a mental image of a large or complex graphic image 
presented to him or her. 

Blind people have problems of recording and retrieval. Blind people commonly use 
two methods to record information for subsequent reading: cassette recorder and Braille 
punch. The laptop and notebook computer offers a portable means to record 
information, input using the keyboard, but the keyboard is often smaller than usual, and 
difficult for a touch typist. Also such a computer does not offer a ready means to 
output the information unless it can support speech synthesis and a sound card. The 
weight of a conventional dynamic Braille display, for when tactile output is needed, 
reduces portability considerably. 

Deafblind people have problems communicating with one another. There is a small 
proportion of the population of people who are both blind and profoundly deaf, though 
this proportion is growing due to greater numbers reaching old age and greater numbers 
of extremely premature babies surviving despite functional defects. Deafblind people 
have to rely on their sense of touch, typically using finger spelling for inter-personal 
communication, and using keyboard and dynamic Braille display for computer input 
and output respectively. They suffer the same problems with Braille as mentioned 
above, and several alternative systems have been tried, as follows. 

A recent development comprises a virtual reality "data glove" which can identify 
characters being spelt out by the user to send text (computer input), coupled with a 
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"hand-tapper" which can tap a pattern related to finger spelling on the hand receiving 
the text (computer output). The glove is expensive to produce, and there would only 
appear to be a very small market for this product. 

A device known as Dexter is a robot hand. It has been developed to simulate a hand 
signing with a deafblind manual alphabet, allowing the receipt of text. However 
deafblind people use subtle feeling of movements on the surface of the hand to help 
them to "read" the characters, and the robot hand has proved unsatisfactory. 
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i peopie nave problems communicating with others, outside their own circle ox 
community. Very few people know tactile sign language. There are several ways for 
people without knowledge of tactile sign language to communicate with a deafblind 
person: they may talk to an intervenor, who types the text for the deafblind person to 
read on a dynamic Braille display; or they might type directly themselves; or they might 
talk to an intervenor who signs in tactile language to the deafblind person, hi the 
absence of an intervenor or the above equipment, they may draw letters on the hand of 
the deafblind person. If the deafblind person is unable to respond orally, then there 
needs to be equipment or a convenor for this direction of communication. In any case, 
conversation is very slow, and suitable conditions may be needed for conversation to ' 
take place at all. 



For remote communication, conversation is via textual message interchange. The 
system known as Hasicom is an example of a system to support such communication 
over the telephone. 



Deafblind people have problems in obtaining adequate education. Current methods of 
teaching deafblind people are extremely labour intensive, and many deafblind people 
have poor literacy skills. Computer aided teaching systems lack a means of immediate 
feedback to the pupil in the tactile domain. The majority of the deafblind people 
acquired their second sensory impairment through gradual deterioration. However, 
even so, there is a reluctance to learn Braille before absolutely necessary, by which time 
it is more difficult because of a lack of feedback between teacher and pupil. 

Deaf people have problems relying on sight only. There are various situations in which 
a deaf person may be trying to lip-read and watch the speaker's expression and body 
language, while at the same time follow the text of what is being said. The text might 
be provided as subtitles, or by a stenographer (or palantypist). For example, at 
meetings a deaf person may have a stenographer who types in phonetic code, which is 
translated into English and presented as text on a lap-top screen in front of the deaf 
person. 



Alternatively there may be an interpreter providing a sign-language translation of the 
speaker's words. In either case the deaf person has a split visual attention. 

Dyslexic people have problems using a single sensory mode, namely their sight. It has 
been estimated that as much as 10% of the population have some form of dyslexia, and 
have difficulty with reading printed text because of this. They are often helped by 
multi-modal display. 
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Visually impaired people have problems with computer and information access. The 
trend towards the electronic office and the information society offers the potential of 
equal ease of access to information, and equal ease of manipulation of that information. 
However, people with visual impairment find it increasingly difficult to access 
computers in the home and in the workplace, as the fashion for visual graphics grows, 
and software tools become more diverse, more feature-laden, and more visually 
complex, employing overlapping windows, icons and a mouse. A means of finding 
one's way around a screen is becoming increasingly urgent for people with visual 
impairment 

The solution generally adopted by people with severe visual impairment is based on 
"screen-readers" which can be set to "look" at certain areas of the screen, and read out 
any new text appearing using speech synthesis or dynamic Braille display. The screen- 
readers have had to become more complex as the visual interface becomes more 
visually complex with successive versions of commercial windowing systems. This in 
turn has made them more difficult to use. 

A problem with speech output and Braille display concerns the lack of distinction 
between upper and lower case letters. It is easy for a blind person, typing on the 
computer, to sKp into upper case without noticing, or forget to revert to lower case at 
the appropriate time. 

Many computer users risk developing repetitive strain injury (RSI). A problem for 
people who need to spend lengthy periods typing into a computer is the risk of 
repetitive strain injury. Special keyboards have been designed, allowing one to hold 
one's arms in a more "natural" position. 

People with dexterity in only one hand require value a means of rapid single-handed 
input. One solution is to use a conventional keyboard with software to map the set of 
keys of one hand onto the other hand, and use depression of the space bar to indicate a 
character from the mapped set. Another solution is to use chorda] input. 

Deaf people have a problem of relying on sight for speech access. There appears to be 
no prior art in the area of speech representation for the tactile domain, except the 
SPIRAL system, developed at Cambridge University. Most of the prior art in tactile 
devices for the deaf concerns devices which present speech vibration to the skin, and 
are used to supplement lip-reading, so that, for example, a deaf person can be made 
aware of which consonants and vowels are voiced. The SPIRAL system uses advanced 
speech recognition technology, and presents the user with phonemes using an array of 
about 40 solenoids to tap or vibrate onto the hand. 

People who are mute or with speech impediment lack a means of generating fluent 
speech. There appears to be no prior art in the area of direct manual generation of 
speech using chordal input Generating speech indirectly via text has the disadvantage 
that the system becomes language specific. Also expressive content cannot be 
transmitted, as the timing, pitch and stress is lost. 

Blind people lack suitable devices for pointing (analogue input) and pointers (analogue 
output). For many computer applications it is necessary to have a pointing mechanism, 
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which can be used to move the position of a point, the "cursor", on a screen or in 
virtual space. Existing pointing mechanisms include: 

• a mouse; 

• a tracker ball; 

• a joystick; 

• a touch tablet; 

• a motion tracking system. 

A motion tracking system can track the motion of the hand in two or more degrees of 
movement. One such system determines the position of the hand relative to the 
person's body, by measuring the change in distance, between a device on the hand and 
two (or three) points on the person's body, typically using acoustic or electromagnetic 
Doppler devices. Another such system measures the induced current resulting from 
motion of a conducting wire attached to the hand either in the earth's magnetic field, or 
in the field of a nearby magnet 

A disadvantage of these mechanisms is that, without a cursor and screen, there is 
limited feedback to the user to indicate to which point the cursor has been moved. 



According to the present invention there is provided a tactile system including a module 
adapted to be brought into contact with the skin of a user of the system, and having an 
array of pressure sensitive input transducers and an array of pressure generating output 
transducers, a central processing unit and an interface adapted to allow signals 
representative of a character generated by the input transducers to be entered into a data 
store in the central processing unit and operating signals from the data store 
representative of a character to be applied to the output transducers. 

The present invention can include a communication system, to allow communication of 
signals associated with input and output between different locations. Thus characters 
generated as input at one location can be presented as output at another location, and 
vice versa. 

The present invention will normally include a sound output system, adapted to operate 
in conjunction with the input and output transducers, but it can be used alone. 
Alternatively, or in addition, the invention can include a visual display system, likewise 
adapted to operate in conjunction with the input and output transducers. The sound 
output system can include a speech synthesiser for indicating orally characters or words 
being entered into or retrieved from the data store. The visual display system can 
display characters or words being entered into or retrieved from the data store. The 
sound output and/or visual display can be synchronised with the tactile input/output. 

The tactile system allows both input to a computer and output from a computer. 
Typically the invention includes keys, similar to those on a conventional keyboard, for 
input, and "buttons" (round-ended pins) for output, see for example Figure 2. Each 
key has an associated transducer which converts pressure applied by the user on the key 
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into an electric signal. Each button has an associated transducer which converts an 
electric signal into a force transmitted via the button onto the skin of the user. 

Input and output can be discrete or analogue. Discrete means that a device, such as a 
key or a button with its associated transducer, can be in one of two states, off or on, 
whereas analogue allows a variation (continuous or in steps) between these states. In the 
analogue form of the present invention, the input and output transducers have a range 
of operating states, the degree of movement of the tactile surface of the transducers 
being related to a feature of the characters being entered into or retrieved from the data 
store. 

Typically the arrangement of keys and/or buttons is symmetrical so that the mechanism 
can be used by either left or right hand, with each pattern for one hand being a mirror 
image of the pattern for the other hand. The line of symmetry 33 is shown for the 
example in Figure 2. For this ambidextrous arrangement there can be a switch, for 
setting by the user according to left or right hand use. 

The buttons for output can be activated by transducers, typically solenoids, either in 
sequence or simultaneously to produce patterns of stimulation on the skin. The pattern 
can be based on a text-based or a phonetically-based code. The text-based code can be 
derived from Braille, or it can be designed for a particular embodiment of the invention, 
as where the code for consonants is related to the shape of lower case characters. 

For text output, one character pattern is displayed at a time, stimulating neural receptors 
(typically in the palm of the hand of the user) over an area of skin with sufficient force 
to ensure detection. The speed at which the pattern is displayed can be controlled by 
the user, using the tactile input mechanism in control mode. Each button can be 
vibrated against the skin, for example by applying a sinusoidal voltage to the associated 
transducer. The frequency of vibration can be varied, for example to distinguish 
different subsets of characters. Upper case can be distinguished from lower case by 
this means. The energy level in the transducers can be varied, so that for the degree of 
generated pressure or the amplitude of vibration can be varied. The interface between 
the data store and the output transducers is adapted to produce operating signals having 
a plurality of distinct frequencies and/or amplitudes, the different frequencies and/or 
amplitudes of the operating signals being indicative of features of the characters to be 
communicated to a user of the system. 

Typically there is an array of buttons in a hexagonal arrangement, see Figure 1. The 
array is used to simulate the drawing of characters on the skin, by varying the locus of 
vibration over an area within the polygon formed by outer buttons, such that the locus 
traces out the shape of character. The locus of vibration, to be felt on the user's skin, is 
a point between three nearest buttons, determined by die balance of energy of vibration 
of the buttons. This is analogous to a colour triangle where the colour at any point is 
determined by the balance of the primary colours to be seen at the vertices. 
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The invention can include a particular input mechanism. This input mechanism 
produces signals to the computer, analogous to those produced by a conventional 
keyboard and mouse, which allow: 

(i) commands to the computer operating system or application; 

(ii) control of parameters such as cursor position and the speed of output; 

(iii) input of characters of a text-based or phonetically-based code. 

This mechanism can be described as a chordal keypad. The layout is arranged so that 
keys can be depressed to produce patterns, either as chords or sequences, 
corresponding to the output patterns. Typically the arrangement is symmetrical, so that 
the mechanism can be used by either left or right hand, as for the output mechanism. A 
switch serves to indicate left or right hand use. If input and output mechanisms are 
incorporated into a hand-set, as for example see Figure 2, a single switch suffices to 
cover both input and output patterns. 

In an analogue form of the input mechanism, used as a pointing means, the degree of 
depression of each key is measured by a transducer, converting displacement into an 
electrical signal indicating the amount of the displacement. The locus of depression can 
be determined from the degree of depression of adjacent keys. Typically the keys are 
grouped around a central point A user digit (finger or thumb) can slide over the keys, 
depressing one or more keys at once and to varying degrees. The locus of depression 
corresponds to the position of the user's digit at any moment. This locus affects the 
component of the position, or movement of position, of a cursor in the direction of the 
locus from the central point. 

The present invention can include mechanisms where the output transducers are also 
capable of acting as input transducers. Thus the same button can act both for key input 
and tactile output. A depression of the button can either operate a switch or force a 
movement of a pin through a solenoid such as to induce a current which can be 
detected as a key depression. The same solenoid can press and/or vibrate the button 
against the skin for tactile stimulation. Thus an array of buttons can be used for both 
input and output. 

In a similar way, the mechanism for pointing (input) can be used as a pointer (output). 
In this case the output buttons are used as keys for pointing. The degree of depression, 
or speed of depression, of a button is detected by a transducer for input. The degree of 
vibration of the button can be varied using a transducer which can be the same 
transducer as detects input. 

The present invention can include another form of pointer (output) mechanism, where 
transducers can tilt a surface in such a way that the tilt, and its degree and direction, can 
be detected by the user. Typically there is a platform with a pattern of holes which can 
be moved by transducers relative to a plate with a corresponding pattern of 
protuberances, such that some of these protuberances can be made to protrude through 
the holes. The plate is curved relative to the platform, such that only a few of the 
protuberances can protrude at any one time. The tilt of the platform relative to the 
plate, or vice versa, can be judged by the user who feels which protuberances are proud 
of the platform surface, and by how much. 
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The same mechanism can be used as a pointing (input) mechanism, where the user can 
depress and tilt the platform, and the degree of depression and tilting is detected by 
transducers. The platform mechanism can serve for both pointing (input) and pointer 
(output) at the same time. The platform is moved by the user for pointing (input), and 
by the transducers as a pointer (output). 

There is an option for a pointing mechanism, including, but not limited to, a mouse, a 
touch tablet, a joystick, a position tracking device, or a special mechanism based on 
input keys or platform and part of this invention. Figure 3 shows the incorporation of a 
mouse mechanism, 37. 

The present invention can include a speech encoder adapted to encode speech into data 
signals representative of characters to be reproduced by the output transducers as 
patterns of stimuli on the skin. 

The present invention can include a speech encoder operating in response to patterns of 
pressure applied by a user on input transducers, and producing signals which can be 
reproduced as speech-like sound or reproduced as characters on output transducers. 

The input and output mechanisms can be incorporated into a handset. There can be a 
strap over the back of the hand to hold the handset in position, or the mechanisms can 
be incorporated in a glove. For computer operation there is a link to the computer to 
convey signals in each direction. This link can be a cable connector. 

The input, output and pointing mechanisms of the system can be physically separated - 
for example the output mechanism can be worn on the upper arm while the input 
mechanism is mounted on a mouse or joystick used as pointing mechanism. 



The invention will now be described and explained with reference to the accompanying 
drawings, in which: 

• Figure 1 shows in plan an arrangement of pressure generating transducers included 
in embodiments of die invention; 

• Figure 2 shows in plan an arrangement of pressure sensitive and pressure generating 
transducers utilised in embodiments of die invention; 

• Figure 3 is a side view of a hand-held communication module incorporating the 
invention, including a mouse-based pointing means; 

• Figure 4 shows a diagrammatic cross-section of another communication module 
embodying the invention. 

Referring to figure 1, one effect of near simultaneous tapping or vibration at two 
neighbouring points on the skin is to produce a sensation of movement of the 
stimulation across the skin between the points. This effect is exploited by placing a 
number of transducers (e.g. solenoids with pins to tap or vibrate the skin surface) in an 
array, and activate successive transducers one by one to trace out a pattern in time and 
space akin to a hand-written letter being drawn on the hand. This pattern is a three- 
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dimensional code, one dimension being time (like Morse code), and two dimensions 
being space (as with printed characters). 

The patterns for each character can be chosen to be distinctive. With patterns involving 
any number of successive stimulations, there is an unlimited number of patterns to 
choose from, and a set of distinctive patterns can easily be chosen. Some restrictions in 
the form of rules can limit the choice of patterns, but help the user in recognition of 
successive patterns at speed. 



In this particular embodiment, the output is performed by seven solenoids packed in a 
hexagonal arrangement, to stimulate the skin, e.g on palm of the hand. This is shown 
approximately to scale (but buttons should be rounded), in figure 1 . The central 
button, 10, is surrounded by a six buttons, 1 1 to 16, forming a hexagon. 

Vowels are abbreviated to a single button operation. Other characters of the alphabet 
have a sequential pattern, simulating the drawing of a lower case character in a stylised 
manner. The English alphabet, as written onto the right hand with palm down covering 
the buttons of figure 1, is output as follows: 



i 
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1^ 10; 16 
H 13^10,16 
10, 13, 14, 12 



tiTTf* h^A** 16 ^' 1 ab ° Ut &e ^ ° f symmetiy ' see 33 « fig" 1 * 2, for writing 



There are four character subsets: lower case letters, capital letters, numbers and special 
characters. These subsets are distinguished by each having a characteristic frequency of 
vibration. Changes between these four subsets are indicated by shift characters- "shift 
to lower case", -shift to caps", "shift to numbers", and "shift to special". A fifth subset, 
which comprises space character, punctuation characters and the shift characters, is not 
distinguished by frequency: 



colon 



• v- 

:-c- 



i 
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Capital letters have the same patterns as lower case letters. The number subset has 
number patterns as follows: 




These patterns are chosen to correspond as far as possible to the use of a telephone 
keypad arrangement for input. Mathematical symbols are included in this subset. 



Timing is based on a unit which can be varied by the user, in the range from 10 
milliseconds to 200 milliseconds, in approximately logarithmic steps over that range. 
There is one unit between the operation of successive buttons of a letter pattern. There 
is a two to five unit pause between letter patterns of a word. There is a space character 
(single operation of lower middle button, 12) to separate words. 



As a reader becomes more fluent, the speed of presentation is increased. At some point 
the reader can change to having the whole pattern presented simultaneously, i.e. with all 
buttons operated at the same time. 



Amplitude can also be varied by the user. Normally the amplitude will be set to the 
lowest value for which the user can reliably sense die characters, in order to minimise 
power consumption. 



Referring to figure 2 - there is shown a combined input/output module. The input 
mechanism and the output mechanism are mounted together on a handset shaped like a 
mouse. The layout of the input keys is shown in the top half of figure 2. It is arranged 
so that patterns can be keyed corresponding to the patterns for output on the buttons 
shown in the bottom half of figure 2. The patterns are as described above in reference 
to figure 1. Key 20 corresponds to buttonlO, 21 to 11, and so on. If operated by the 
right hand, the index (first) finger could press keys 23 and/or 24, the middle (second) 
finger could press 25 and/or 20, the ring (third) finger could press 26, the litde (fourth) 
finger could press 27, and the thumb would press 22. Note that, rather than having 22 
implemented as shown in figure 2, it can be implemented as two separate keys, each in 
a plane angled to finger-operated keys, on opposite sides of the handset, see figure 3. 
This allows for natural movements of the fingers and the thumb, for a relaxed hand. 
The edges of the keypad plane are marked at 31 and 32. There is an angle of 
approximately 60° between the plane of the finger keys and the output buttons, forming 
a ridge, see 30 on figures 2 and 3, so that while the output buttons are against the palm 
of the hand, the finger knuckles are comfortably bent around 34 for operating the keys 
with a relaxed hand. 



Not shown in the figure is a switch for switching between left- and right-handed 
operation. Also not shown is a switch for switching the keypad operation between 
input mode, command mode and control mode. When in command mode, the keys are 
used to move a text cursor, and perform edit commands. When in control mode, the 
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keys are used for control of output, including adjustment of speed, pausing, repeating 
characters, repeating of words. 

In one form of the input mechanism, the keys are arranged as on a telephone keypad. 
For numerical input, the keys can be used singly as they would be on a normal 
telephone. Key 25 is number 2 in the middle of the top row; keys 24, 20 and 26 are in 
the next row as numbers 4, 5 and 6; keys 23 and 21 are in the third row as numbers 7 
and 9; and key 22 is in the bottom row as number 0. The pattern for output is made to 
correspond where possible, see table above. 

In an embodiment of the tactile system, for use with graphics and spatial information, 
the input and output mechanisms are mounted on a ''mouse", that is, a handset with a 
ball whose motion is tracked, see 37 in figure 3, so that it can be used as a pointing 
mechanism. Digital representations of graphic images or virtual surfaces are stored in 
computer memory. The mouse pointing mechanism is used to move a cursor to 
indicate a position on the graphic image or virtual surface, just as a mouse is used to 
move a cursor position on a screen. An array of tactile output buttons is activated 
according to the characteristics of the image in the vicinity of the cursor. The tactile 
system allows the user to find the position of point objects on the image or surface, to 
follow lines (e.g. indicating roads, boundaries or contours on a map), and to add 
markings to the image or surface (e.g. a route on the map). The type and identity of an 
object can be displayed to the user, visually (in large print for partially sighted), in audio 
(with speech synthesiser or with digitised speech) or using the tactile system's character 
display. There is the same 7 button arrangement as shown in figure 1, which can be 
used for text output, e.g. of place names on a map, as well as for exploring the graphic 
image or virtual surface. It also has the same key arrangement for input as described 
previously. Buttons and keys are used as described previously. However there is some 
additional capability for both output and input, as now to be described. 

There are two output modes: one for characters and one for navigating the image or 
surface. While in command mode, one of the keys is used Kke a mouse button, e.g. for 
drawing onto die graphic image or virtual surface. While in navigation mode, the 
buttons are activated as the cursor is moved. When the cursor is near an object in the 
image or surface, outer buttons of the array nearest the object vibrates at a rate and/or 
amplitude dependent on the distance to the object or its boundary. The centre button 
vibrates according to what is at the point on the map under the cureor. 

Suppose the user moves the cursor towards a straight line on the map. On approaching 
the straight line, the user senses at first the proximity and direction of the nearest part of 
the line. As the Kne is reached, the centre button is activated, and the approximate 
direction of the line is indicated by vibrations of the outer buttons most closely aligned 
with the direction. This allows the user to stop the cursor on the line, and then move it 
along the line to follow a boundary or the shape of an object 

One embodiment, which can be used for graphics and spatial information but can also 
be used in virtual reality applications, is the same as the previously described 
embodiment, but has the keys and buttons mounted on a hand-set without mouse ball, 
and uses a different mechanism for pointing. The movement of the handset, or of the 
hand holding the hand-set, is tracked in two or three dimensions. The buttons are used 
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to sense the proximity and direction of the nearest objects in the virtual space. With 
two dimensions and 7 buttons, the sensing of directions is as in the previous 
embodiment With three dimensions and 7 buttons, the outer pairs of opposite buttons 
represent the x, y and z axes. Buttons 1 1, 16 and 15 represent x, y and z positive; 
buttons 14, 13 and 12 represent x, y and z negative. Directions cannot be judged as 
accurately as was possible in two dimensions, since these primary directions are 90° 
apart, whereas the primary directions for the hexagonal arrangement in two dimensions 
are 60° apart 

The tactile output mechanism of the current invention can be incorporated into any 
system which needs to display information. Such systems include appliances in the 
home, lifts to tell you which floor you are on, and kiosks in public places. The buttons 
are set into the display panel, as in figure 1. In such a system, the output might be 
combined with the invention's tactile input mechanism or with a conventional input 
mechanism. A pointing device could be included for access to maps in public places. 
However for this application the mouse and virtual reality approaches of the previously 
described embodiments are probably inappropriate. 

Referring to figure 4, a tilting tactile pointer mechanism is shown. In this example a 
platform, 40, moves, and a plate, 43, is fixed. (In another form the platform is fixed 
and the plate moves.) The corners of the platform rest on four pins, 41, which can be 
moved by solenoids, 42, such that the platform, 40, can be moved and tilted an amount 
determined by the relative activation of each pin, 41, corresponding to the direct current 
in each solenoid, 42. Each pin, 41, is attached at one end to the platform, 40, in such a 
way as to allow the platform, 40, to tilt, and at the other to a soft spring (not shown in 
the figure), such that at rest the platform, 40, is suspended just above the plate. In 
order that the user's finger or thumb, resting on the platform, 40, can feel the extent of 
tilting, In the platform there are small holes, 45, in a hexagonal pattern (as in figure 1 
but with the holes close together), through which protuberances, 44, protrude, 
according to the movement and tilting of the platform, 40, relative to the plate, 43. 

The same mechanism is used for pointing, using induction in the solenoids, 47, from 
movement of the pins, 41, when the platform, 40, is depressed, to determine the point 
and degree of depression. It is necessary that the platform, 40, at rest is a little above 
the fixed plate, 43, and that the curvature of the plate, 43, is such that when the 
platform, 40, is gradually depressed at any particular point, the platform, 40, eventually 
touches the plate at that same point 

A speech-based form of the current invention allows direct translation of spoken word 
between the sound domain and the tactile domain. For speech to touch, the invention 
uses transducers to tap or vibrate and produce patterns of stimulation, typically on the 
palm side of the hand, and the patterns correspond to phonetic characteristics of the 
speech, recognised using the front-end of a real-time speech recognition system. 

In an embodiment of the speech-based form of the current invention, an array of 
buttons, stimulating the tips of the fingers, is used for consonants and each consonant 
has a discrete pattern. A second array of buttons, stimulating the palm of the hand, is 
used for vowels, and any vowel sound will have a locus of vibration (according to the 
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formants) which moves continuously if the vowel sound changes during enunciation, as 
for a diphthong. 

Vowel sounds are mapped onto points and curves within a diagram whose co-ordinates 
represent different formants. The extremes of vowels conventionally foim the comers 
of a quadrilateral in this 'Vowel space". The invention maps vowels onto a hexagonal 
space using the front end of a speech recognition system. The invention utilises an 
interpolation effect whereby, if nearby points on the skin are stimulated simultaneously, 
die stimulus may be perceived to be between those points. The locus of perceived 
stimulus can be moved continuously in two dimensions within a triangle of three 
buttons, by changing the energy distribution between the buttons. The hexagonal 
arrangement is effectively a grouping of six such triangles, and thus allows the locus to 
be moved anywhere within the hexagon formed by the outer buttons. Vowel sounds 
are mapped to points and curves within this hexagon- Variations in pitch and stress can 
be captured by varying the frequency and amplitude of vibration of buttons. 

For touch to speech, the invention allows patterns to be generated using combinations 
of finger movements, in the fashion of chords. These chords are translated into a 
phonetic stream. This stream comprises a sequence of character codes, each 
representing the phonetic characteristics of successive vowels and consonants. Timing, 
stress, and pitch information is included in the stream. The stream can be transmitted 
to a recipient person (the "listener") where it can be processed by the back-end of a 
speech synthesiser to produce speech in real-time. With the options for timing, stress, 
and pitch, it is possible to convey an expressive content of speech. 

For the generation of vowel sounds, a pointing mechanism based on a platform is 
operated by thumb, as in figure 4 The thumb can point to a point, or trace out a curve, 
in vowel space, to generate a pure vowel or a diphthong respectively. 

This embodiment of present invention uses the same speech recognition technology as 
SPIRAL, but presents the phoneme stream in a different way, using patterns to present 
phonetic speech content rather than individual pins to represent phonemes. The 
presentation of vowels using a quasi-analogue means differs from SPIRAL'S strictly 
discrete means. 

This embodiment differs from those which are text-based in that: 

• the coding is phonetic rather than being based on a textual characters; 

• the timing is synchronised with speech; 

• there is a greater number of buttons and keys; 

• there are buttons to be felt by the fingers; 

• an analogue mechanism is needed for vowels. 

For remote communication between a speaking person and a deaf person, the fust stage 
of translation, involving speech recognition, has to take place at the speaker's location, 
and the intermediate phonetic representation of the speaker's utterances has to be 
encoded and transmitted over the telecommunication network for decoding and tactile 
display to the deafbKnd person. In the reverse direction, the tactile speech has to be 
encoded and transmitted, for decoding and speech synthesis for the hearing person. 
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The speech processing and coding at the speaker's location could be performed by a 
software package, without the need for an instance of the invention itself. Note that the 
encoded speech could be sent in packets over the Internet, as in an "Internet Phone" 
facility. 

In this embodiment an array of buttons is used for input and output according to 
whether the system is in input or output mode, so that the fingers of one hand can both 
key in and be stimulated by the tactile output. A platform is used for pointing by the 
thumb, as in figure 4, but the corresponding pointer output is provided by buttons 
stimulating the palm of the hand, as in figure 1. 



There are many benefits of the invention. 

Features for reading text have the benefit that the invention provides inexpensive tactile 
output of text using a single cell with typically seven buttons (or pins) which can be 
made to stimulate the skin using cheap off-the-shelf solenoids. The buttons can be 
several times the size and spacing of embossed Braille dots, or the pins used in a 
conventional dynamic Braille display, so the sensitivity problem is avoided. 
Furthermore the buttons can be vibrated, and adding vibration (optimally at around 
200Hz) improves the detection of the stimulus by particularly sensitive receptor neurons 
present in the glabrous skin on the palm side of the hand. 

For ease of learning, especially for elderly people, the buttons can be operated in such a 
way as to simulate the writing of lower-case characters on the palm of the hand. The 
invention can employ seven buttons as being the minimum for efficient simulation of 
handwriting. (Six buttons are arranged in a hexagon with the seventh button at the 
centre. An effect of near simultaneous tapping or vibration at two neighbouring points 
on the skin is to produce a sensation of movement of the stimulation across the skin 
between the points. For handwriting simulation, the locus is moved from button to 
button of the pattern.) As the user gains familiarity, the drawing time can be reduced 
until the pattern is presented as simultaneous stimuli to the hand, allowing rapid reading, 
with speeds comparable to those achieved by fluent readers of Braille. By having 
different vibration frequencies, a user can distinguish between upper and lower case 
characters, and does not have to rely on remembering the last change between upper 
and lower, as one has to with 6-dot Braille. 

The invention provides a virtual display of the graphic surface, over which a pointer can 
be moved and characteristics of its location sensed using the same array of pins or 
buttons as for character output. A hexagonal arrangement of seven pins is a minimum 
sufficient to allow the user to examine the surface efficiently. (The central pin indicates 
to the user whether there is an object under the cursor. Six outer pins suffice to 
indicate the direction of a line object under the cursor. When there is no object under 
the cursor, they can indicate die direction of a nearby object, so the cursor can be 
moved towards it Thus the entire image can be explored efficiently.) 

For portability, a form of the invention combines input and tactile output in a single, 
light-weight, hand-held unit. This unit, used in conjunction with a laptop or palmtop 
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computer, provides a highly portable system for recording and retrieval, with many 
advantages over a tape recorder. 

A feature of particular to deafblind people is that the invention can be used to 
communicate text in purely tactile domain. An overlapping two-way conversation is 
possible, as it is possible to send at the same time as receiving. The invention can be 
used for local or remote communication with a deafblind person. The invention can be 
used by both parties in communication, thus obviating the need for a keyboard or a 
dynamic Braille display. Each party can be standing or walking while communicating, 
using a hand-held unit. 

There are features for feedback and reinforcement. The invention allows tactile input 
with immediate feedback, in visual, audio or tactile domain, to reinforce learning and 
provide correction when necessary. Communication between teacher and pupil is 
improved, in the situations of both special and mainstream education. 
The invention can be realised with extra output buttons to make the simulation of 
writing on the hand more realistic, and allow the beginner to gradually move towards 
the stylised code. For people with at least partial vision, the tactile output can be 
displayed on a screen at the same time as being output, so the locus of vibration can be 
followed visually. 

There are features to complement sight for deaf people. The invention allows text to 
be presented in tactile modality, to complement sight. The invention allows speech to 
be presented in tactile modality, to complement sight. The speech-based form of the 
system allows direct translation of spoken word into the tactile domain. 

The invention can be used by dyslexics. The tactile display can reinforce the visual 
display of text, and therefore help with reading. This is especially valuable for deaf 
dyslexics for whom audio reinforcement cannot be used. 

The invention can be incorporated into a consumer product or public facility to provide 
tactile output in a standard form for those who need it The input mechanism of the 
invention can be incorporated into a consumer product or public facility, to replace or 
supplement an existing form of input, and provide the user with tactile input in a 
standard form. 

As regards access to die computer screen for people with visual impairment, the 
invention can be used to explore the layout on a screen, treating the areas as graphic 
elements, see above. The content of an area can be read out using a speech synthesiser, 
as for a normal screen-reader, or it can be read out using the invention's tactile output. 

The invention includes chordal input designed such that strain is reduced because the 
hand is in a natural shape, and the handset can be operated in any orientation. 

The invention includes chordal input designed such that the system can be operated 
entirely by one hand. The input and output arrangement is typically symmetrical, such 
that the system can be operated by either hand alone. 
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A form of the invention includes mechanisms, based on the phonetics of speech, which 
translate the spoken word directly between the sound domain and the tactile domain, 
without an intermediate natural language textual form. A real-time speech recognition 
system recognises the phonetic characteristics of speech sounds within a fraction of a 
second; and this data is presented in the tactile domain, fast enough to be effectively 
synchronous with lip movements of the speaker. Thus the deaf person can feel the 
spoken word while watching the speaker. Because this version of the invention is based 
on universal phonetic characteristics of vowels and consonants, it can be used for 
communication in any spoken language in the world. Distinctive dialects can be 
recognised by the shift the vowels in vowel space. 

The invention allows for speech generation, of particular benefit to people who are 
mute or with speech impediment. The invention allows patterns to be generated using 
combinations of finger and thumb movements, in the fashion of chords, these chords 
being translated into a phonetic stream, which can be processed by the back-end of a 
speech synthesiser to produce speech in real-time. The user can produce a distinctive 
dialect "accent" by shifting the vowels in vowel space. Hand movement, in two of the 
six degrees of movement, can represent stress and pitch, to add expressive content to 
the generated speech. The pitch of vowels for tonal languages can be handled in this 
way, for example by relating pitch to sideways movement of the hand or to twist of the 
hand. 

The invention includes features for pointing and for pointers, of particular use to blind 
people for access to graphic information. The features can also be employed in the 
speech-based form of the invention, as for example in pointing to a vowel in vowel 
space. (For pointing, die invention's chordal input mechanism is used in analogue 
mode, or a tilting platform is used For pointing, the invention includes pointer 
mechanisms corresponding to the above pointing mechanisms.) A pointer and its 
corresponding pointing mechanism can be combined into a single composite 
mechanism, which reduces size and weight of the system. 
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CLAIMS 

1. A tactile communication system including a module adapted 
to be brought into contact with the skin of a user of the system 
and having an array of pressure sensitive input transducers and 
an array of pressure generating output transducers , a central 
processing unit and an interface adapted to allow signals 
representative of a character generated by the pressure sensitive 
input transducers to be entered into a data store in the central 
processing unit and operating signals from the data store 
representative of a character to be applied to the pressure 
output transducers. 

2. A system as claimed in claim 1, including a speech 
synthesiser adapted to operate in conjunction with the output 
transducers . 

3- A system as claimed in claim 1 or claim 2, in which a speech 
synthesiser is adapted to operate in conjunction with the input 
transducers . 

4. A system as claimed in any preceding claim, including means 
for indicating visually characters being entered into or 
retrieved from the data store. 

5. A system as claimed in any preceding claim, in which the 
input or output transducers can be operated sequentially. 

6. A system as claimed in any preceding claim in which the 
output transducers are adapted to produce vibratory output 
signals . 

7. A system as claimed in claim 6, in which the frequency 
and/or amplitude of the vibrations of the output transducers can 
be varied. 
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8. A system as claimed in claim 7, in which the interface 
between the data store and the output transducers is adapted to 
produce operating signals having a plurality of distinct 
frequencies and or amplitudes, the different frequencies and/or 
amplitudes of the operating signals being indicative of features 
of the characters to be communicated to a user of the system. 

9. A system as claimed in any preceding claim, in which the 
input and output transducers have only two operating states. 

10. A system as claimed in any of claims 1 to 8, in which the 
input and output transducers have a range of operating states, 
the degree of movement of the tactile surfaces of the transducers 
being related to a feature of the characters being entered into 
or retrieved from the data store. 

11. A system as claimed in any preceding claim, in which the 
characters reproduced by the output transducers approximate to 
alpha-numeric characters. 

12. A system as claimed in any of claims 1 to 10, in which the 
characters reproduced by the output transducers approximate to 
Bra i 1 1 e characters . 

13. A system as claimed in any preceding claim, wherein the 
module includes two sets of arrays of input and output 
transducers, one set being the mirror image of the other so that 
the module can be used with either hand. 

14. A system as claimed in any of claims 6 to 8 or 10 to 13, 
wherein a position on the surface on which the tactile surfaces 
of the output transducers lie can be determined by varying the 
energy of the vibrations of the three output transducers nearest 
to that position to give weighted mean vibrations at the said 
position. 
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15. A system as claimed in claim 10, in which a position on the 
surface on which the tactile surface of the input transducers lie 
is determined from the degree of depression of three input 
transducers surrounding that position. 

16. A system as claimed in any preceding claim, in which the 
output transducers are also are capable of acting as input 
transducers . 

17. A system as claimed in any preceding claim, including a 
speech encoder adapted to encode speech into data signals 
representative of characters to be reproduced by the output 
transducers . 

18. A system as claimed in claim 17, in which there is included 
a transmission system for the data signals to be reproduced by 
the output transducers. 

19. A system as claimed in claim 17 or claim 18, wherein the 
speech encoder operates in response to patterns input to the 
speech encoder via the input transducers. 

20. A tactile communication system substantially as hereinbefore 
described with reference to and as shown in the accompanying 
drawings . 
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